{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# fast-neptune\n",
    "\n",
    "> Quickly track your Jupyter Notebook experiments with NeptuneML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fast-neptune` is a library that helps you quickly record all the information you need to launch your experiments, when you are using Jupyter Notebooks. Reproducibility has become a crucial issue in Machine Learning, not only for research, but also for real world applications, where we want to have robust results, and track every set of parameters tested, along with their results.\n",
    "\n",
    "`fast-neptune` spirit is highly inspired from [nbdev](http://nbdev.fast.ai/) in the user experience, so it is recommended to have a look first at it.\n",
    "\n",
    "Furthermore, `fast-neptune` is built upon the solution [neptune-ml](https://neptune.ai/) which allows users to quickly record on a public or private repository. Following the [quick introduction](https://docs.neptune.ai/python-api/introduction.html) is a plus to understand `fast-neptune`, but as we will see it's quite intuitive."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fast-neptune` is available on pypi so you can simply run :"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pip install fast-neptune`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to use"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`fast-neptune` has implemend several features to help you when you run ML experiments.\n",
    "These include :\n",
    "<ul>\n",
    "    <li>metadata about the machine where the code is run, including OS, and OS version</li>\n",
    "    <li>requirements of the notebook where the experiments are run</li>\n",
    "    <li>parameters used during the experience, which means the names of the values of the variables you want to track</li>\n",
    "    <li>code you used during the run that you want to record</li>\n",
    "</ul>\n",
    "\n",
    "Note that code and parameters are not mandatorily tracked, while the two former are, though the whole purpose of this library is to facilitate the tracking of parameters and code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metadata "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Metadata is tracked about the requirements used and information about the Python version and OS used.\n",
    "\n",
    "They are added automatically when an experiment is created, through the functions [`get_metadata`](/core#get_metadata) and [`create_requirements`](/core#create_requirements)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Property"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Properties refer to variables you want to record. \n",
    "\n",
    "To record properties, simply add `#property` to each cell containing the variables you want to record.\n",
    "\n",
    "Note : if one variable in a property is a file encapsulated in a `Path` object, it is automatically tracked, and this file will be sent to NeptuneML. You can disable this option when creating the experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Code\n",
    "\n",
    "Code cells can also be tracked.\n",
    "\n",
    "To do so, simply add `#code` to each cell you want to record. You have the possibility to specify under what name the cell code will be registered. By default, all cells with `#code` will be added chronogically to the file \"main.py\", but if you can put them in specific files by adding the name of the module you want them to be put in.\n",
    "\n",
    "Example : add `#code dataloading.py` to the cells that take care of the data loading."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example with fastai vision to train a MNIST classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand more how all of this fit in, we will follow a simple example, using fastai library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we will log in using NeptuneML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Neptune login\n",
    "from neptune.sessions import Session\n",
    "import getpass\n",
    "\n",
    "# First we get the token\n",
    "api_token = getpass.getpass(\"Please enter your NeptuneML API token : \")\n",
    "session = Session(api_token=api_token)\n",
    "\n",
    "# Then we enter user name and project\n",
    "user_name = \"danywin\"\n",
    "project_name = \"fast-neptune\"\n",
    "\n",
    "project = session.get_project(project_qualified_name=f'{user_name}/{project_name}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will start using `fast-neptune`.\n",
    "\n",
    "First we will record the code used to load the data and learner. To do so, we simply add `#code` to the cell used to load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fast_neptune.core import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#code dataloading.py\n",
    "from fastai.vision import *\n",
    "\n",
    "path = untar_data(URLs.MNIST_SAMPLE)\n",
    "data = ImageDataBunch.from_folder(path)\n",
    "learn = cnn_learner(data, models.resnet18, metrics=accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we want to keep track of the number of epochs during training, and the learning rate we used. To do so we add `#property` to the cell we want to track."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#property\n",
    "n_epoch = 1\n",
    "lr = 1e-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we use [`fast_experiment`](/core#fast_experiment) to create our experiment. Note that you must pass it the name of the notebook you are using for your experiments, and the global variables you are currently using to record each variable in the properties cells.\n",
    "\n",
    "Here we will create an experiment, then record the validation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "https://ui.neptune.ai/danywin/fast-neptune/e/FAS1-18\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.052504</td>\n",
       "      <td>0.014153</td>\n",
       "      <td>0.994603</td>\n",
       "      <td>00:04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#code\n",
    "nb_name = \"index.ipynb\"\n",
    "globs = globals()\n",
    "\n",
    "with fast_experiment(project,nb_name,globs) as exp:\n",
    "    learn.fit_one_cycle(n_epoch,lr)\n",
    "    pred,y = learn.get_preds()\n",
    "    \n",
    "    score = accuracy(pred,y).item()\n",
    "    exp.send_metric(\"valid_accuracy\",score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now if we go on the link provided by NeptuneML we can have a closer look at this experiment :\n",
    "\n",
    "Here we can see the parameters that were registered, including epoch, learning rate, OS, and Python version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"./imgs/parameters.PNG\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the source code tab, we can have a look at the bits of code we registered, here the dataloading.\n",
    "<img src=\"./imgs/code.PNG\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the artifacts we also have the modules used in this notebook with their version : \n",
    "<img src=\"./imgs/requirements.PNG\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we logged the validation accuracy, which can be found in the Logs tab : \n",
    "<img src=\"./imgs/valid_score.PNG\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
